Assist thread for injecting cache memory in a microprocessor

ABSTRACT

A data processing system includes a microprocessor having access to multiple levels of cache memories. The microprocessor executes a main thread compiled from a source code object. The system includes a processor for executing an assist thread also derived from the source code object. The assist thread includes memory reference instructions of the main thread and only those arithmetic instructions required to resolve the memory reference instructions. A scheduler configured to schedule the assist thread in conjunction with the corresponding execution thread is configured to execute the assist thread ahead of the execution thread by a determinable threshold such as the number of main processor cycles or the number of code instructions. The assist thread may execute in the main processor or in a dedicated assist processor that makes direct memory accesses to one of the lower level cache memory elements.

This invention was made with government support under NBCHCO20056awarded by PERCS phase one. THE GOVERNMENT HAS CERTAIN RIGHTS IN THISINVENTION.

BACKGROUND

1. Field of the Present Invention

The present invention is in the field of microprocessors and moreparticularly microprocessors employing multiple levels of cache memoryto reduce memory access latency.

2. History of Related Art

Memory latency refers to the delay associated with retrieving data frommemory in a microprocessor-based data processing system. The pace atwhich microprocessor cycle times have decreased has exceed improvementsin memory access times. Accordingly, memory latency has remained as abarrier to improved performance and has increased in significance witheach additional advance in microprocessor performance.

Numerous techniques, varying widely in both effectiveness andcomplexity, have been proposed and/or implemented to reduce performancebottlenecks attributable to memory latency. Perhaps the most significantand pervasive technique is the use of cache memory. A cache memory is astorage element that is relatively small and fast compared to systemmemory. The cache memory contains, at any time, a subset of the datastored in the system memory. When a general purpose microprocessorrequires data, it attempts to retrieve the data from its cache memory.If the needed data is not currently present in the cache memory, thedata is retrieved from system memory and the contents of the cachememory are updated at the same time that the data is provided to themicroprocessor. In this manner, the cache memory is continuously beingupdated with the most recently accessed data.

The effectiveness of cache memory in addressing system memory latency isdependent upon a high percentage of memory accesses being fulfilled fromthe cache memory. Fortunately, studies have shown that most programstend to exhibit spatial and temporal locality in their memory accesspatterns. Spatial locality implies that programs tend to access datathat is nearby (in terms of memory address) data that was recentlyaccessed. Temporal locality implies that programs tend to access datathat was recently accessed. Both factors validate the use of cachememory subsystems to address memory latency.

Cache memory is so effective in reducing latency that cache memorysubsystems have evolved rapidly in both size and architecture. Typicalcache memory subsystems now include multiple levels of cache memoryunits that are tiered to provide a spectrum of size and speedcombinations. Referring to FIG. 1, for example, selected elements of aconventional microprocessor-based data processing system 100 aredepicted to illustrate the use of cache memory. In FIG. 1 system 100includes a central processing unit 102 and three tiers of cache memorybetween the microprocessor 102 and system memory 110. A level one (L1)cache 104 is the smallest, fastest, and most expensive cache memory unitof the three. L1 cache 104 sits “next” to central processing unit (CPU)102 and is the first cache memory accessed by CPU 102. If a CPU memoryaccess can be satisfied from the contents of L1 cache 104, latency isminimized to perhaps two CPU cycles.

When a CPU memory access “misses” in L1 cache 104 (i.e., CPU 102attempts to access data that is not present or valid in L1 cache 104)the memory request is passed to the larger and slower L2 cache 106 todetermine if the requested data is valid therein. If the memory access“hits” in L2 cache 106, the data is retrieved to satisfy the CPU requestand the L1 cache is updated with the requested data. If the memoryaccess misses in L2 cache 106, the memory request is passed to the stilllarger and slower L3 cache 108. If the memory access hits in L3 cache108, the data is retrieved and provided to CPU 102 and the contents ofL2 cache 106 and L1 cache 104 are updated. Finally, if a memory accessmisses in L3 cache 108, the data is retrieved from system memory 110 andeach cache memory 104, 106, and 108 is updated.

The latency associated with L1 cache 104 is usually capable of being“hidden” using techniques such as prefetching, multithreaded execution,out of order execution, speculative execution, and the like. Thesetechniques, unfortunately, typically require sophisticated hardware thatconsumes valuable microprocessor real estate. Moreover, such techniquesare not capable of hiding long latencies associated with lower levelcache miss events. It would be desirable, therefore, to implement asystem and method for reducing latency in multiple-tiered cache memorysubsystems. It would be further desirable if the implemented solutiondid not require a significant amount of dedicated hardware and reliedinstead, on existing hardware and architectures to the greatest extentpossible.

SUMMARY OF THE INVENTION

The identified objectives are addressed in the present invention by adata processing system that includes a main processor havingconventional access to multiple levels of cache memories. The mainprocessor executes a main thread compiled from a source code object. Thesystem may also include a dedicated assist processor for executing anassist thread also derived from the source code object. The assistthread includes memory reference instructions of the main thread andonly those arithmetic instructions required to resolve the memoryreference instructions. A scheduler configured to schedule the assistthread in conjunction with the corresponding execution thread isconfigured to execute the assist thread ahead of the execution thread bya determinable threshold such as a specific number of programinstructions (wherein the assist thread is constrained to execute nomore than a specified number of instructions ahead of the main thread),or a specific number of memory access operations (wherein the assistthread is constrained to have no more than a specified number ofoutstanding memory access misses). The assist thread may execute in themain processor or in the assist processor. The assist processor, ifused, is preferably designed to be architecturally adjacent to one ofthe lower level caches so that the assist processor can make directaccesses to one of the lower level caches.

BRIEF DESCRIPTION OF THE DRAWINGS

Other objects and advantages of the invention will become apparent uponreading the following detailed description and upon reference to theaccompanying drawings in which:

FIG. 1 is a block diagram of selected elements of a memory subsystem fora general purpose microprocessor according to the prior art;

FIG. 2 is a conceptual illustration of a main thread and an assistthread according to an embodiment of the present invention;

FIG. 3 is a block diagram of selected elements of a microprocessor-baseddata processing system according to an embodiment of the presentinvention;

FIG. 4 is a flow diagram of a method and software code for reducingmemory latency according to an embodiment of the present invention; and

FIG. 5 is a diagram illustrating the creation of a main thread and anassociated assist thread according to the present invention.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Itshould be understood, however, that the drawings and detaileddescription presented herein are not intended to limit the invention tothe particular embodiment disclosed, but on the contrary, the intentionis to cover all modifications, equivalents, and alternatives fallingwithin the spirit and scope of the present invention as defined by theappended claims.

DETAILED DESCRIPTION OF THE INVENTION

Generally speaking, the invention is concerned with reducing memoryaccess latency in a microprocessor-based computing device or dataprocessing system without substantially adding to the hardware elementsthat are already present. When a piece of source code is compiled tocreate an executable “thread” referred to herein as the main thread, asecondary executable file, referred to herein as the assist thread orsimply the assist thread is also created. In another implementation, theassist thread is created as a section of the main thread so that it doesnot have to be a separate file. The assist thread is designed to executeahead of the main thread for the sole purpose of making memory accessreferences that the main thread will be making when it executes shortlythereafter. When the assist thread makes a memory access reference, thecache subsystem responds by determining whether the referenced memoryaddress is valid in the cache and, if not, by retrieving data from thereferenced system memory address. When the main thread subsequentlyaccesses the same memory address, a cache miss is much less likely. Byleveraging the cache subsystem's native ability to update itself inresponse to a cache miss, the use of an assist thread reduces cachemisses without requiring any significant additional or dedicatedhardware.

Referring now to FIG. 2, a conceptual illustration of a main thread 201and a corresponding assist thread 202 according to one implementation ispresented to emphasize significant aspects of assist thread 202.Generally, assist thread 202 is a streamlined version of main thread 201designed to make all of the same memory references that main thread 201will make. More specifically, assist thread 202 as shown in FIG. 2includes only the following instructions from main thread 201: thememory access instructions (or at least all of the load instructions) ofmain thread 201, those arithmetic instructions required to resolve thereferences in a memory access instruction, and the control flow (branch)instructions required to determine whether a memory access instructionmust be executed. All other arithmetic, floating point, and otherinstructions from main thread 201 are eliminated from assist thread 202to minimize its size and complexity. By scheduling the assist thread inconjunction with the main thread and constraining the two threads suchthat the assist thread executes “ahead” of the main thread, the numberof cache miss events encountered by the main thread as it executes isimproved significantly.

As depicted in FIG. 2, for example, main thread 201 includes six memoryaccess instructions and two arithmetic instructions. The memory accessinstructions include load instructions 210, 212, 214, and 215, storeinstructions 213 and 217 and arithmetic instructions 211 and 216. Loadinstructions 210, 214, and 215 are direct memory references that can beresolved from the load instruction itself. Load instruction 212, on theother hand, is an indirect memory reference that cannot be resolvedwithout determining the value stored in R1, which is dependent on thearithmetic add instruction 211.

The assist thread 202 that corresponds to main thread 201 includes loadinstructions 220, 222, 224, and 225 corresponding to the main threadload instructions 210, 212, 214, and 215 respectively and an arithmeticadd instruction 221 corresponding to main thread add instruction 211.Thus, whereas main thread 201 includes a total of eight instructions,assist thread 202 includes only five instructions. The add instruction221 is included in assist thread 202 because it is needed to resolve thememory reference made in load instruction 222. The assist thread 202does not include an add instruction corresponding to main thread addinstruction 216 because there is no subsequent load instruction thatdepends on the outcome of add instruction 216.

The depicted implementation of assist thread 202 does not includeinstructions corresponding to the store instructions 213 and 217 of mainthread 201. The store instructions of main thread 201 are omitted fromassist thread 202 because, as a general rule, latency is a lesssignificant issue for store instructions than it is for loadinstructions. Unlike load instructions, store results can be bufferedand handled (written back to memory) without substantially impactingmain processor performance. In other implementations, however, assistthread 202 may include all memory reference instructions includinginstructions that will prefetch the memory locations corresponding tothe store instructions in the execution thread. Such an implementationmight be suitable, for example, with respect to an application thatproduces more store instructions than the cache subsystem is capable ofbuffering.

Referring to FIG. 5 momentarily, assist thread 202 is created from thesame source code object 502 that is used to create main thread 201. Aconventional compiler 504 receives source code object 502 as its inputand, in response, creates main thread 201 using compiling techniquesthat are known in the field. In addition, however, source code object502 is also provided to a program slicer 506 that creates an assistthread 202 suitable for execution in conjunction with main thread 201.

Program slicer 506 may include elements of developmental andcommercially distributed object code slicers such as the WisconsinProgram Slicing Tool described at www.cs.wisc.edu/wpis/slicing_tool/ andthe CodeSurfer tool distributed by GrammaTech, Inc. Such object codeslicers enable one to find meaningful decompositions of programs, wherethe decompositions consist of elements that are not textuallycontiguous. A program slice with respect to a set of program elements Sis a projection of the program that includes only program elements thatmight affect (either directly or indirectly) the values of the variablesused at members of S. In the context of the present invention, programslicer 506 is used to extract memory reference instructions andarithmetic instructions upon which the memory reference instructionsdepend.

Referring now to FIG. 3, a block diagram of selected elements of a dataprocessing system 300 according to one embodiment of the presentinvention is depicted. In the depicted embodiment, system 300 includes amain processor 302, an L1 cache 304, an L2 cache 306, and an L3 cache308. In addition, system 300 includes a dedicated, special purposeprocessor referred to in herein as assist processor 320. Assistprocessor 320, in the depicted embodiment, is directly connected orarchitecturally adjacent to a lower level cache memory unit, namely, L2cache 306. A scheduler 305 is connected to main processor 302 and toassist processor 320. An instruction cache 330 provides instructions tomain processor 302 and assist processor 320.

Main processor 302 may be composed of multiple individual processingdevices and each individual processing device may have multipleprocessing cores. Assist processor 320 may be integrated into the samechip as main processor 302. In embodiments emphasizing the ability ofassist processor 320 to access lower cache levels directly, however,assist processor 320 is preferably implemented at the same chip as thatof a lower level cache. Assist processor 320 preferably includes alimited set of the elements found in main processor 302, such as aload/store unit and an arithmetic logic unit, but preferably omits otherelements, such as a floating point unit, to conserve spatial and powerrequirements. Scheduler 305 might be included on the same chip as mainprocessor 302 or implemented alongside the assist processor 320.Scheduler 305 may also be a software construct, whose functions areexecuted by the main execution thread and the assist thread.

Scheduler 305 is configured to control execution of main thread 201 inconjunction with assist thread 202. Specifically, scheduler 305 isconfigured to schedule assist thread 202 to execute ahead of main thread201 by a predetermined interval. The interval may be measured in termsof (approximated) main processor cycles or in terms of codeinstructions. Ideally, scheduler 305 maintains execution of assistthread 202 sufficiently ahead of main thread 201 such that the cachesubsystem response to a cache miss experienced by assist thread 202produces a cache hit in main thread 201 when the main thread executes amemory access to the same memory address. On the other hand, it isdesirable to maintain execution of the two threads sufficiently close toeach other to prevent intervening programs from invalidating data thatwas referenced by assist thread 202. Thus, there may be an optimum delaybetween assist thread 202 and main thread 201 and scheduler 305 thatminimizes cache miss events experienced by main thread 201.

In one embodiment, scheduler 305 is implemented entirely in softwarewith the main execution thread periodically executing “marker”instructions that serve to inform the assist thread where it is in theexecution stream. These “marker” instructions could be implemented asstores to a special address that the assist thread polls. Bysynchronizing with the main thread in this manner, the assist thread canbe constrained to be ahead of the main thread within a specifiedinstruction execution interval. By providing a limited form of hardwaresupport, the “marker” instructions can also be implemented as writes toa special register that the assist thread polls. In another embodiment,scheduler 305 is implemented purely in the assist thread by forcing itto have a limit on the number of outstanding memory access misses. In athird embodiment, the main execution thread increments a counter eachtime it has a cache miss. The assist thread also increments a countereach time it encounters a cache miss. The assist thread issues memoryaccesses only as long as the difference between the two counters iswithin a specified limit.

As described above, assist thread 202 generates the same memory accessreferences that main thread 201 will make when it executes. When theassist thread 202 memory references are made by assist processor 320,the cache subsystem (the aggregation of L1 cache 304, L2 cache 306, L2cache 308, and the corresponding control logic) will cause the requesteddata to be injected into the cache subsystem. When main thread 201executes a memory access instruction, presumably shortly after assistthread 202 executes the corresponding instruction, the memory accessmade by main thread 201 is unlikely to result in a cache miss. Even ifit does result in a cache miss, the main thread's access is not likelyto suffer the full cache miss penalty because the assist thread hasalready issued the memory access.

The embodiment depicted in FIG. 3 emphasizes a loose coupling betweenmain thread 201 (FIG. 2), which executes on main processor 302 andassist thread 202, which executes on assist processor 320. The depictedembodiment is referred to as loose coupling between assist thread 202and main thread 201 because assist thread 202 executes on assistprocessor 320 and main thread 201 executes on main processor 302. Loosecoupling between the two threads beneficially enables main processor 302additional processing capacity and thus represents a better use ofresources at the cost of added complexity in controlling the schedulingof the two threads.

In contrast, another embodiment of the invention uses an assist threadthat is tightly coupled to its corresponding main thread. In one suchembodiment, for example, main processor 302 supports simultaneousmultithreaded (SMT) execution. In this embodiment, main processor 302includes two (or possibly more) SMT “slots” or virtual processors thatcan execute essentially simultaneously but have their own independentcontext. SMT processors recognize that, even in the absence of longdelays caused by cache misses and other events, processor resources tendto be underutilized meaning that, at any given time, there are stageswithin the execution units (load/store units, arithmetic units, etc.)that are vacant. The POWER5™ processor from IBM Corporation is anexample of a general purpose processor that supports SMT.

In the tightly coupled embodiment, main thread 201 and assist thread 202execute on a common physical processor, such as main processor 302. Inperhaps the most tightly coupled implementation, main thread 201 andassist thread 202 execute on an SMT microprocessor where each thread isallocated an SMT slot of the processor. The tightly coupled embodimentbeneficially simplifies the logic and/or code needed to control thedelay between the two threads at the cost of consuming some of thegeneral purpose processing capability of main processor 302 for aprogram (i.e., assist thread 202) that consists largely of “dead code”(code that does not alter the contents of memory).

Returning momentarily to the loosely coupled implementation depicted inFIG. 3, an advantage of the depicted implementation is the placement ofassist processor 320 where it is architecturally adjacent to the lowerlevel cache memory elements. Specifically, assist processor 320 asdepicted in FIG. 3 is configured to issue memory access referencesdirectly to L2 cache 306. This architecture is beneficial because itprevents assist processor 320 from making a large number of L1 accesses.It is desirable to keep the L1 cache 304 available for the main threads.By requiring the L1 cache to be write-through (as is the case with mostprocessors such as the IBM Power4 and Power5), this configuration alsopermits the assist thread to become aware of writes to memory from themain thread. More specifically, when the assist processor 320 isarchitecturally adjacent to a lower level cache, it is placed at thecoherence point for the system.

Assist processor 320 and the cache subsystem may also be configuredwhere cache references to the L2 cache 306 made by assist processor 320“trickle up” to L1 cache 304. In other words, a reference made directlyto L2 cache memory 306 by assist processor 320 updates the contents ofL1 cache 304. This technique is an extension of existing techniques inwhich memory access references are always issued to the L1 cacheinitially and, when misses occur, the data is taken from the L2 cache ifit is present and valid, the L3 cache if the data is not valid in the L2cache, and so forth.

Portions of the invention may be implemented as a set or sequence ofcomputer executable instructions (i.e., computer software code) storedon a computer readable medium such as a hard disk, floppy diskette, CD,magnetic tape and the like. During execution of the computer code,portions of the code may reside in a volatile storage element such as asystem memory or a cache memory.

Referring now to FIG. 4 a flow diagram illustrates selected elements orsteps of a method 400 for reducing cache miss events in a dataprocessing system according to one embodiment of the present inventionis depicted. Method 400 is also a conceptual depiction of computer codethat, when executed, performs the depicted elements or steps.

In the depicted embodiment, the method is initiated by deriving or“slicing” (block 402) an assist thread from a source code object. Thesource code object from which the assist thread is sliced is also thesource code object that is compiled to generate a main thread with whichthe assist thread is associated. The assist thread is subsequentlyscheduled for execution (block 404) in conjunction with the execution ofthe main thread. As described above, the assist thread may be scheduledfor execution on the assist processor or on the main processor itself.The scheduler is configured to constrain (block 406) the execution ofthe assist thread relative to the main thread so that the assist threadremains ahead of the main thread. The delay between the two threads maybe based on a specific number of program instructions (wherein theassist thread is constrained to execute no more than a specified numberof instructions ahead of the main thread), or may be based on a specificnumber of memory access operations (wherein the assist thread isconstrained to have no more than a specified number of outstandingmemory access misses).

The remaining elements of method 400 depicted in FIG. 4 refer to assistthread instructions and the processor on which the assist thread isexecuting (whether that processor is the main processor or the assistprocessor). For purposes of describing FIG. 4, the assist threadinstructions are presumed to be executing on the assist processor. Afterscheduling and properly constraining the relative execution of the mainthread and the assist thread, an assist thread instruction is issued(block 408) for execution.

Initially, the assist processor determines (block 410) whether theissued instruction provides a value that is required by a subsequentinstruction. If no subsequent assist thread instruction relies ordepends on a value generated or retrieved by the issued assist threadinstruction (block 412), the issued assist thread instruction is treatedas a prefetch instruction wherein the assist processor does not awaitcompletion of the issued assist thread instruction before proceeding toissue a subsequent assist thread instruction. If, however, the issuedassist thread instruction does retrieve, or generate a value that isneeded by a subsequent assist thread instruction, the assist threaddetermines (block 414) whether the issued assist thread instructionmisses in the cache subsystem.

As described earlier the assist processor may be architecturallyadjacent to the L2 cache (or to the cache level at which coherence ismaintained in the system). In such cases, the assist processor mayaccess the cache to which it is adjacent so that, for example, insteadof initially determining whether a memory reference hits in the L1cache, the assist processor may access the L2 (or lower) cache and makea hit/miss determination based on the contents of the lower level cache.Regardless of the specific cache level that the assist processoraccesses directly, if the assist processor determines in block 414 thata memory reference produces a cache hit, the assist processor accordingto the depicted embodiment of method 400 executes (block 416) the issuedinstruction and awaits completion of the instruction before proceeding.

If, however, the issued assist thread instruction includes a memoryreference that misses in the cache memory, the depicted embodiment ofmethod 400 includes executing (block 418) the instruction based on apredetermined policy. The policy preferably indicates which of a set ofassist thread instruction execution strategies is pursued when an assistthread instruction having dependent assist thread instructions misses inthe cache. The alternatives might include, for example, (1) issuing theassist thread instruction and waiting for the result before proceedingwith the assist thread, (2) predicting the result of the assist threadinstruction (i.e., predicting the value of the data that would beretrieved) and resuming execution without subsequently checking theaccuracy of the prediction, and (3) predicting the result of the assistthread instruction, resuming execution, and subsequently verifying theprediction and taking corrective action if the prediction was incorrect.In the case of a policy involving prediction of the data to beretrieved, any of a variety of prediction mechanisms may be suitablyemployed. An example of a data prediction algorithm, desirable for itssimplicity, is the “predict 0” algorithm in which the value of anyparticular piece of data is predicted to be 0 based on evidencesuggesting that a large percentage of retrieved data has a value of 0.Another example for predicting the contents of a particular memorylocation is the “use last” prediction mechanism in which a stale (nolonger valid) value in a cache memory is used as the predicted value forthe corresponding memory address.

It will be apparent to those skilled in the art having the benefit ofthis disclosure that the present invention contemplates a system andmethod for improving performance and reducing cache miss penalties in adata processing system. It is understood that the form of the inventionshown and described in the detailed description and the drawings are tobe taken merely as presently preferred examples. It is intended that thefollowing claims be interpreted broadly to embrace all the variations ofthe preferred embodiments disclosed.

1. A computer program product comprising: a non-transitory computerreadable storage medium that stores program code for reducing latency ina data processing system having a multi-level cache hierarchy includinga higher level cache having a lower access latency and a lower levelcache having a higher access latency, wherein the program code causesthe data processing to perform: generating a main thread and an assistthread from a source code object, wherein the assist thread includes asmaller subset of the instructions in the main thread; scheduling theassist thread to execute in conjunction with the main thread;constraining execution of the main thread relative to the assist thread,such that each instruction in the main thread having a correspondinginstruction in the assist thread executes after the correspondinginstruction in the assist thread; making a memory access requested by aninstruction of the assist thread directly to the lower level cachewithout first accessing the higher level cache and making acorresponding memory access requested by an instruction in the mainthread to the higher level cache; and responsive to a memory accessindicated by a selected memory reference instruction in the assistthread missing in the lower level cache, determining whether asubsequent assist thread instruction depends on the selected memoryreference instruction and, if so, taking action based on an assistthread policy, wherein the assist thread policy is selected from a setof policies consisting of awaiting completion of the selected memoryreference instruction, predicting an outcome of the selected memoryreference instruction, and predicting the outcome of the selected memoryreference instruction and verifying the outcome upon completion.
 2. Thecomputer program product of claim 1, said generating comprisingcompiling the source code object to create the main thread and compilingselected instructions of the source code object to derive the assistthread.
 3. The computer program product of claim 1, wherein theconstraining comprises constraining the main thread to execute apredetermined number of main processor cycles behind the assist thread.4. The computer program product of claim 1, wherein the constrainingcomprises constraining the main thread to execute a predetermined numberof code instructions behind the assist thread.
 5. The computer programproduct of claim 1, wherein the scheduling comprises scheduling theassist thread to execute in a first simultaneous multithreading (SMT)slot of a main processor of the data processing system and schedulingthe main thread to execute in a second SMT slot of the main processor.6. The computer program product of claim 1, wherein the schedulingcomprises scheduling the assist thread to execute on an assist processorof the data processing system and scheduling the main thread to executeon a physically distinct main processor of the data processing system.7. The computer program product of claim 1, wherein the subset ofinstructions includes: memory access instructions, those arithmeticinstructions required to resolve memory references in the memory accessinstructions, and any branch instruction required to determine whether amemory access instruction among the memory access instructions is to beexecuted.
 8. The computer program product of claim 1, wherein theprogram code causes the data processing system to execute the assistthread executes such that the assist thread has no more than a specifiednumber of outstanding memory access misses.
 9. A method for reducingmemory latency associated with a main thread in a data processing systemincluding a higher level cache having a lower access latency and a lowerlevel cache having a higher access latency, the method comprising:generating a main thread and an assist thread from a source code object,wherein the assist thread includes a smaller subset of the instructionsin the main thread; scheduling the assist thread to execute inconjunction with the main thread; constraining execution of the mainthread relative to the assist thread, such that each instruction in themain thread having a corresponding instruction in the assist threadexecutes after the corresponding instruction in the assist thread;making a memory access requested by an instruction of the assist threaddirectly to the lower level cache without first accessing the higherlevel cache and making a corresponding memory access requested by aninstruction in the main thread to the higher level cache; and responsiveto a selected memory reference instruction in the assist thread missingin the lower level cache memory of the system, determining whether asubsequent assist thread instruction depends on the selected memoryreference instruction and, if so, taking action based on an assistthread policy, wherein the assist thread policy is selected from a setof policies consisting of awaiting completion of the selected memoryreference instruction, predicting an outcome of the selected memoryreference instruction, and predicting the outcome of the selected memoryreference instruction and verifying the outcome upon completion.
 10. Themethod of claim 9, wherein said generating comprises compiling a sourcecode object to create the main thread and compiling selectedinstructions of the source code object to derive the assist thread. 11.The method of claim 9, wherein constraining execution of the main threadrelative to the assist thread comprises constraining the main thread toexecute a predetermined number of main processor cycles behind theassist thread.
 12. The method of claim 9, wherein constraining executionof the main thread relative to the assist thread comprises constrainingthe main thread to execute a predetermined number of code instructionsbehind the assist thread.
 13. The method of claim 9, wherein thescheduling comprises scheduling the assist thread to execute in a firstsimultaneous multithreading (SMT) slot of a main processor of the dataprocessing system and scheduling the main thread to execute in a secondSMT slot of the main processor.
 14. The method of claim 9, wherein thescheduling comprises scheduling the assist thread to execute on adedicated assist processor and scheduling the main thread to execute ona physically distinct main processor of the data processing system. 15.The method of claim 9, wherein the subset of instructions includes:memory access instructions, those arithmetic instructions required toresolve memory references in the memory access instructions, and anybranch instruction required to determine whether a memory accessinstruction among the memory access instructions is to be executed. 16.The method of claim 9, and further comprising executing the assistthread such that the assist thread has no more than a specified numberof outstanding memory access misses.
 17. A data processing system,comprising: a multi-level cache memory hierarchy including at least ahigher level cache having a lower access latency and a lower level cachehaving a higher access latency; a main processor coupled to access themulti-level cache memory, wherein the main processor simultaneouslyexecutes a main thread and an assist thread, wherein the assist threadis derived from a same source code object as the main thread andcontains a smaller subset of the instructions in the main thread, andwherein the main processor makes memory accesses indicated by the mainthread to the higher level cache and makes memory accesses indicated bythe assist thread directly to the lower level cache without firstaccessing the higher level cache, wherein responsive to a selectedmemory reference instruction in the assist thread missing in the lowerlevel cache memory of the system, the main processor determines whethera subsequent assist thread instruction depends on the selected memoryreference instruction and, if so, takes action based on an assist threadpolicy, wherein the assist thread policy is selected from a set ofpolicies consisting of awaiting completion of the selected memoreference instruction, predicting an outcome of the selected memoryreference instruction, and predicting the outcome of the selected memoryreference instruction and verifying the outcome upon completion; and ascheduler that schedules execution of the assist thread and the mainthread for simultaneous execution, wherein execution of the main threadis constrained such that each instruction in the main thread having acorresponding instruction in the assist thread executes after thecorresponding instruction in the assist thread.
 18. The system of claim17, wherein the scheduler schedules the assist thread to execute no morethan a specified number of instructions ahead of the main thread. 19.The system of claim 17, wherein the main processor executes such thatthe assist thread has no more than a specified number of outstandingmemory access misses.
 20. The system of claim 17, wherein: the mainprocessor supports at least a first simultaneous multithreading (SMT)slot in which the main thread executes and a second SMT slot in whichthe main thread executes.
 21. The system of claim 17, wherein the subsetof instructions includes: memory access instructions, those arithmeticinstructions required to resolve memory references in the memory accessinstructions, and any branch instruction required to determine whether amemory access instruction among the memory access instructions is to beexecuted.
 22. A data processing system, comprising: a multi-level cachememory hierarchy including at least a higher level cache having a loweraccess latency and a lower level cache having a higher access latency; amain processor coupled to the multi-level memory hierarchy, wherein themain processor executes a main thread and makes memory accessesrequested by instructions in the main thread to the higher level cache;an assist processor physical distinct from the main processor, whereinthe assist processor executes an assist thread derived from a samesource code object as the main thread and contains a smaller subset ofthe instructions in the main thread, and wherein the assist processormakes memory accesses requested by instructions in the assist threaddirectly to the lower level cache without first accessing the higherlevel cache, wherein responsive to a selected memory referenceinstruction in the assist thread missing in the lower level cache memoryof the system, the assist processor determines whether a subsequentassist thread instruction depends on the selected memory referenceinstruction and, if so, takes action based on an assist thread policy,wherein the assist thread policy is selected from a set of policiesconsisting of awaiting completion of the selected memory referenceinstruction, predicting an outcome of the selected memory referenceinstruction, and predicting the outcome of the selected memory referenceinstruction and verifying the outcome upon completion; and a schedulerthat schedules execution of the assist thread and the main thread forsimultaneous execution, wherein execution of the main thread isconstrained such that each instruction in the main thread having acorresponding instruction in the assist thread executes after thecorresponding instruction in the assist thread.
 23. The system of claim22, wherein the scheduler schedules the assist thread to execute no morethan a specified number of instructions ahead of the main thread. 24.The system of claim 22, wherein the assist processor executes such thatthe assist thread has no more than a specified number of outstandingmemory access misses.
 25. The system of claim 22, wherein the subset ofinstructions includes: memory access instructions, those arithmeticinstructions required to resolve memory references in the memory accessinstructions, and any branch instruction required to determine whether amemory access instruction among the memory access instructions is to beexecuted.